Rebalancing the Load in Distributed File Systems

Distributed file systems are key building blocks for cloud computing applications supported the Map Reduce programming paradigm. In such file systems, nodes at the same time serve computing and storage functions; a file is partitioned off into variety of chunks allotted in distinct nodes so Map Reduce tasks are often performed in parallel over the nodes. However, in a cloud computing setting, failure is that the norm and nodes is also upgraded, replaced, and accessorial within the system. Files can even be dynamically created, deleted, and appended. This leads to load imbalance during a distributed file system; that's, the file chunks don't seem to be distributed as uniformly as doable among the nodes. Rising distributed file systems in production systems powerfully depend upon a central node for chunk reallocation. This dependence is clearly inadequate during a large-scale, failure-prone setting as a result of the central load balancer is anesthetize significant employment that's linearly scaled with the system size, and should therefore become the performance bottleneck and therefore the single purpose of failure. During this paper, a completely distributed load rebalancing formula is bestowed to cope with the load imbalance drawback. Our formula is compared against a centralized approach during a production system and a competitor distributed answer bestowed within the literature. The simulation results indicate that our proposal is comparable the prevailing centralized approach and significantly outperforms the previous distributed formula in terms of load imbalance issue, movement price, and algorithmic overhead. The performance of our proposal enforced within the Hadoop distributed filing system is additional investigated in a cluster setting. Index Words: Distributed files, map reduce, replaced, dependence, chunk reallocation.